Two-channel-based high-availability

ABSTRACT

A method for providing two-channel-based high-availability in a cluster of nodes is disclosed. In an embodiment, a method comprises: initiating, by a local control plane executing on a first node, a first state for an underlay control channel and a second state for a management control channel; detecting a bidirectional forwarding detection (“BFD”) control packet from a second node; determining whether the BFD control packet has been received from the underlay control channel; in response to determining that the BFD control packet was received from the underlay control channel: parsing the BFD control packet to extract a first diagnostic code; updating the first state with the first diagnostic code; determining whether both the first state and the second state indicate that the second node is unreachable; in response to determining that the second node is unreachable, initiating a switchover of services configured on the second node.

BACKGROUND

A high availability system is a system that is resilient to failures ofthe system's components. Typically, this is achieved by providingredundant components so that if one component fails, a redundantcomponent can take over performing the tasks of the failed component.

HA devices, such as edge nodes, may be grouped into clusters. The nodesin a cluster may work as a team to provide services even if some of thenodes fail. As long as at least one of the nodes in a cluster remainsactive, the cluster may provide the services configured on the nodes.Examples of the services may include load balancing, traffic forwarding,data packet processing, VPN services, DNS services, and the like.

Nodes in a cluster may operate in either an active mode or a standbymode. If a node in a cluster fails, then, if possible, a surviving nodeassumes an active role and provides the services that were configured onthe failed node.

Unfortunately, detecting failures of nodes in node clusters is ofteninefficient and difficult. Typically, HA nodes in a cluster communicatewith each other via Bidirectional Forwarding Detection (“BFD’) channels.However, since the BFD channel may be configured with an aggressivetimer, relying on communications exchanged via the BFD channel may leadto false detections of failures. For example, when no response isreceived to three consecutive packets sent to a node, an aggressivetimer may flag failure of the node even if the node is still healthy.This may happen because the BFD traffic is usually communicatedalongside the user traffic over the same channel, and the responses fromthe nodes are lost due to congestion caused by a high-volume usertraffic, not due to the node's failure. Nevertheless, failure to timelydetect BFD control packets from the node may trigger failover even ifthe node is still healthy.

SUMMARY

Techniques are presented herein for providing HA support by a nodecluster. The techniques provide two-channel-based HA that relies oncommunications exchanged via two channels established between hostshosting the nodes of the cluster. The purpose of using two channels,instead of one, is to improve reliability of the HA support. Forexample, if one channel fails, then the system may rely on theinformation obtained via the second channel. The cluster may include apair of edge nodes, one of which operates in an active mode and anotherin a standby mode.

In an embodiment, a pair of channels established between two hosts isconfigured to provide support for BFD-compliant communications. One ofthe channels is referred to as an underlay control channel (or anunderlay channel), while another channel is referred to as a managementcontrol channel (or a management channel). The pair of channel may beimplemented either between virtual network interface cards (“VNICs”) ofthe hosts or between physical network interface cards (“PNICs”) of thehosts.

If the pair of channels are implemented between VNICs, then the BFDcontrol packets communicated via the channels are monitored by localcontrol planes of the respective hosts. If the pair of channels areimplemented between PNICs, then the BFD control packets communicated viathe channels are monitored by local control planes of the operatingsystem (“OS”) of the hosts.

In an embodiment, local control planes monitor BFD control packetscommunicated via both an underlay channel and a management channel. Thelocal control planes may, for example, extract diagnostic codes from theBFD control packets, and use the diagnostic codes to determine whether aneighbor node has failed. For example, if BFD control packets receivedvia either channel indicate that the neighbor node has failed, then theservices configured on the neighbor node may be switched over ontoanother node.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a block diagram depicting an example physical implementationview of an example logical network environment 10 for implementingtwo-channel-based HA for a cluster of nodes.

FIG. 2A is a block diagram depicting an example implementation oftwo-channel-based HA for a cluster of nodes.

FIG. 2B is a block diagram depicting an example implementation oftwo-channel-based HA for a cluster of nodes.

FIG. 3 is a block diagram depicting an example implementation oftwo-channel-based HA in physical network interface cards of hosts.

FIG. 4A is an example flow chart for implementing a two-channel-basedhigh-availability approach.

FIG. 4B is an example flow chart for implementing a two-channel-basedhigh-availability approach.

FIG. 5 is a block diagram depicting an example mandatory section of anexample of a generic BFD control packet.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the method described herein. It will be apparent,however, that the present approach may be practiced without thesespecific details. In some instances, well-known structures and devicesare shown in a block diagram form to avoid unnecessarily obscuring thepresent approach.

1. Example Physical Implementations

FIG. 1 is a block diagram depicting an example physical implementationview of an example logical network environment 10 for implementingtwo-channel-based HA for a cluster of nodes. In the depicted example,environment 10 includes two or more hosts 106A, 106B, and one or morephysical networks 155.

Hosts 106A, 106B are used to implement logical routers, logical switchesand virtual machines (“VMs”). Hosts 106A, 106B are also referred to ascomputing devices, host computers, host devices, physical servers,server systems or physical machines. Each host may be configured tosupport several VMs. In the example depicted in FIG. 1, host 106A isconfigured to support a VM 107A, while host 106B is configured tosupport a VM 107B. Additional VMs may also be supported by hosts106A-106B.

Virtual machines 107A-107B are executed on hosts 106A, 106B,respectively, and are examples of virtualized computing instances orworkloads. A virtualized computing instance may represent an addressabledata compute node or an isolated user space instance. VMs 107A-107B mayimplement edge nodes, edge node gateways, and the like.

Hosts 106A, 106B may also be configured to support execution ofhypervisors 109A and 109B, respectively.

Hypervisors 109A, 109B are software layers or components that supportthe execution of multiple VMs, such as VMs 107A-107B. Hypervisors 109Aand 109B may be configured to implement virtual switches and forwardingtables that facilitate data traffic between VMs 107A-107B. In certainembodiments, virtual switches and other hypervisor components may residein a privileged virtual machine (sometimes referred to as a “DomainZero” or “the root partition”) (not shown). Hypervisors 109A and 109Bmay also maintain mappings between underlying hardware 115A, 115B,respectively, and virtual resources allocated to the respective VMs.

Hardware component 115A may include one or more processors 116A, one ormore memory units 117A, one or more PNICs 118A, and one or more storagedevices 121A.

Hardware component 115B may include one or more processors 116B, one ormore memory units 117B, one or more PNICs 118B, and one or more storagedevices 121B.

2. Example Two-Channel High-Availability Configuration

2.1. Example Vnic-Based Configuration

FIG. 2A is a block diagram depicting an example implementation oftwo-channel-based HA for a cluster of nodes. In the depicted example,hosts 106A-106B may be edge service gateways. Host 106A provides supportfor VNIC 190A and VNIC 190AA, while host 106B provides support for VNIC190B and 190BB. VM 107A supports, among other things, a local controlplane 250A and a data path process 260A. VM 107B supports, among otherthings, a local control plane 250B and a data path process 260B.

Hosts 106A-106B also provide support for execution of hypervisors 109Aand 109B, respectively.

In the depicted example, hardware 115A includes, among other things,PNICs 118A, while hardware 115B includes, among other things, PNICs118B.

In the depicted example, a two-channel-based HA for a cluster of nodesis implemented using a pair of 170A-170B channels: an underlay controlchannel 170A and a management control channel 170B.

In an embodiment, underlay control channel 170A is established between aVNIC 190A and a VNIC 190B, and it is a channel in an underlay networkused to communicate overlay traffic. Management control channel 170B isestablished between VNIC 190AA and VNIC 190BB. Both channels 170A-170Bmay be used to provide two-channel-based HA for nodes, such as VMs107A-107B. Both channels 170A-170B are used to communicate BFD controlpackets.

Local control plane 250A is configured to monitor both channels170A-170B on VM 107A side, while local control plane 250B is configuredto monitor both channels 170A-170B on VM 107B side. For example, localcontrol plane 250A may monitor BFD control packets detected oninterfaces configured for channels 170A-170B to determine whether VM107B executing on host 106B has failed.

In an embodiment, to determine whether VM 106 executing on host 106B hasfailed, local control plane 250A implements the following rules: if noBFD control packets has been received via both channels 170A-170B fromhost 106B after a timeout, then local control plane 250A deduces that VM107A is unreachable, and thus the services configured on VM 107B shouldbe switched from VM 107B onto VM 107A. However, if a BFD control packetfrom at least one of channels 170A-170B from host 106B includes adiagnostic code indicating host 106B is down, then local control plane250A deduces that VM 170B is down, and thus the services configured onVM 107B should be switched from VM 107B onto VM 107A. In othersituations, local control plane 250A deduces that host 106B is up and sois VM 107B, and therefore, no switchover is needed at this time.

Two-channel-based HA may utilize diagnostic codes included in BFDcontrol packets communicated via underlay control channel 170A andmanagement control channel 170B. Diagnostic codes are described indetail in FIG. 5.

An example of a diagnostic code is a code “7,” which indicates an“administrative down” of a node. Local control plane 250A implemented inVM 107A receives a BFD control packet with the diagnostic code “7” whenVM 107B hosted on host 106B enters an administrative-down-state. If VM107B enters an administrative-down-state, then BFD control packets withthat code are most likely to be detected on interfaces of both channels,and therefore, upon receiving such BFD control packets, local controlplane 250A may generate a message or a request to initiate failover.

However, in some situations, a local control plane may determine thatdiagnostic codes included in BFD-compliant control packets detected oninterfaces of the two channels are different. In such situations, if anyof channels 170A-170B communicated a BFD control message indicating thatVM 107B is down, then, upon receiving such a BFD control packet, localcontrol plane 250A deduces that VM 107B is indeed down, and thus localcontrol plane 250A generates a message or a request to initiatefailover.

In some situations, local control plane 250A awaits receiving a BFDcontrol packets from each channel 170A-170B. If no BFD control packetsis received from underlay control channel 170A (or management controlchannel 170B) after a timeout, then local control plane 250A deducesthat either the channel is down or a corresponding VNIC is down. Iflocal control plane 250A does not receive any BFD control packet fromboth channels 170A-170B after a timeout, then local control plane 250Amay deduce that VM 107B is unreachable, and thus VM 107B is down. Inthis situation, local control plane 250A may generate a message or arequest to initiate failover.

Functionalities of local control plane 250B mirror functionalities oflocal control plane 250B. More specifically, local control plane 250Bmay be configured to monitor both channels 170A-170B and based on BFDcontrol packets detected on interfaces configured for channels 170A-170Bon the side of host 106B, determine whether VM 107A executing on host106A has failed.

2.2. Example Vnic-Based Configuration

FIG. 2B is a block diagram depicting an example implementation oftwo-channel-based HA for a cluster of nodes. In the depicted example,hosts 106A-106B are edge service gateways. OS module 135A on host 106Asupports, among other things, a local control plane 450A and a data pathprocess 460A, while OS module 135B on host 106B supports, among otherthings, a local control plane 450B and a data path process 460B.

In the depicted example, a two-channel-based HA for a cluster of nodesis implemented using a pair of 171A-171B channels: an underlay controlchannel 171A and a management control channel 171B.

In an embodiment, underlay control channel 171A is established between aPNIC 118A and a PNIC 118B, and it is a channel in an underlay networkused to communicate overlay traffic. Management control channel 171B isestablished between PNIC 118AA and PNIC 118BB. Both channels 171A-171Bmay be used to provide two-channel-based HA for hosts 106A-106B. Bothchannels 171A-171B are used to communicate BFD control packets.

Local control plane 450A is configured to monitor both channels171A-171B on host 106B side, while local control plane 450B isconfigured to monitor both channels 171A-171B on host 106B side. Forexample, local control plane 450A may monitor BFD control packetsdetected on interfaces configured for channels 171A-171B to determinewhether host 106B has failed.

In an embodiment, to determine whether host 106B executing on host 106Bhas failed, local control plane 450A implements the following rules: ifno BFD control packets have been received via channels 171A-171B fromhost 106B after a timeout, then local control plane 450A deduces thathost 106B is unreachable, and thus the services configured on host 106Bshould be switched from host 106B onto host 106A. However, if a BFDcontrol packet from at least one of channels 171A-171B from host 106Bincludes a diagnostic code indicating that host 106B is down, then localcontrol plane 450A deduces that host 106B is down, and thus the servicesconfigured on host 106B should be switched from host 106B onto host106A. In the remaining situations, local control plane 450A deduces thathost 106B is up, and therefore, no switchover is needed at this time.

Two-channel-based HA may utilize diagnostic codes included in BFDcontrol packets communicated via underlay control channel 171A andmanagement control channel 171B. Diagnostic codes are described indetail in FIG. 5.

An example of a diagnostic code is a code “7,” which indicates an“administrative down” of a node. Local control plane 450A implemented inan OS module 135A receives a BFD control packet with the diagnostic code“7” when host 106B enters an administrative-down-state. If host 106Benters an administrative-down-state, then BFD control packets with thatcode are most likely to be detected on interfaces of both channels.Therefore, upon receiving such a BFD control packets, local controlplane 450A may generate a message or a request to initiate failover.

However, in some situations, local control plane 450A may determine thatdiagnostic codes included in BFD-compliant control packets detected oninterfaces of the two channels are different. In such situations, if anyof channels 171A-171B communicated a BFD control message indicating thathost 106B is down, then, upon receiving such a BFD control message,local control plane 450A deduces that host 106B is indeed down, and thuslocal control plane 450A generates a message or a request to initiatefailover.

In some situations, local control plane 450A awaits receiving a BFDcontrol packet from each channel 171A-171B. If no BFD control packets isreceived from underlay control channel 171A (or management controlchannel 171B) after a timeout, then local control plane 450A deducesthat either the channel is down, or a corresponding host is down. Iflocal control plane 450A does not receive any BFD control packet fromboth channels 171A-171B before a timeout, then local control plane 450Amay deduce that host 106B is unreachable. In this situation, localcontrol plane 450A may generate a message or a request to initiatefailover.

Functionalities of local control plane 450B mirror functionalities oflocal control plane 450B. More specifically, local control plane 450Bmay be configured to monitor both channels 171A-171B and based on BFDcontrol packets detected on interfaces configured for channels 171A-171Bon the side of host 106B, determine whether host 106A has failed.

3. Example Two-Channel High-Availability Configuration

FIG. 3 is a block diagram depicting an example implementation oftwo-channel-based HA in physical network interface cards of hosts.

In the depicted example, a PNIC 418A is configured in hardware 115A,while a PNIC 418B is configured in hardware 115B. Furthermore, a PNIC419A is configured in hardware 115A, while a PNIC 419B is configured inhardware 115B.

Moreover, underlay control channel 171A is established between PNIC 418Aand PNIC 418B, while management control channel 171B is establishedbetween PNIC 419A and PNIC 419B.

Depending on status of data path process 460B, local control plane 450Bmay determine a diagnostic code for data path process 460B. Furthermore,local control plane 450B may encapsulate the diagnostic code in a BFDcontrol packet and copy the BFD control packet on the interface of bothunderlay control channel 171A and management control channel 171B.

Subsequently, local control plane 450A may detect the BFD control packetwith the diagnostic code on the interface of either underlay controlchannel 171A or management control channel 171B, and analyze thediagnostic code. If the code is for example, a diagnostic code “6”, thenlocal control plane 450A may determine that a concatenated path to host106B is down, and thus temporarily unavailable. Subsequently, localcontrol plane 450A may generate a message to initiate failover ofservices configured on host 106B onto host 106A.

Similarly, depending on status of data path process 460A, local controlplane 450A may determine a diagnostic code for data path process 460A.Furthermore, local control plane 450A may encapsulate the diagnosticcode in a BFD control packet and copy the BFD control packet on theinterface of both underlay channel 171A and management channel 171B.

Subsequently, local control plane 450B may detect the BFD control packetwith the diagnostic code on the interface of underlay channel 171A ormanagement channel 171B and analyze the diagnostic code. If the code isfor example, a diagnostic code “6”, then local control plane 450B maydetermine that a concatenated path to host 106A is down, and thustemporarily unavailable. Subsequently, local control plane 450B maygenerate a message to initiate failover of services configured on host106A onto host 106B.

4. Example Workflow

In an embodiment, a two-channel-based HA approach is implemented betweenany two entities of a HA cluster. For example, if a cluster includes twoedge service gateways, then local control planes implemented in the edgeservice gateways cooperate with each other to determine whether any ofthe two edge service gateways has failed. The local control planes mayfor example, exchange BFD control packets via an underlay controlchannel and a management control channel. The BFD control packetsexchanged via the channels may be generated based on differentinformation available to the local control channels. Based on theexchanged information, the local control channels determine whether theedge service gateways are down.

In an embodiment, hosts hosting nodes of an HA cluster may establishtheir underlay control channels and management control channels at aVNIC-level or at a PNIC level.

FIG. 4A is an example flow chart for implementing a two-channelhigh-availability approach. In the depicted example, it is assumed thata cluster comprises two edge service gateways, each hosted on adifferent host, and that an underlay control channel and a managementcontrol channel are established between either corresponding VNICs orPNICs. The channels are configured to facilitate BFD-compliantcommunications.

In step 402, a local control plane executing on a first node initiatesand stores, in memory of the local control plane, for a neighbor nodethe following: a first state (associated with an underlay controlchannel) and a second state (associated with a management controlchannel).

In step 404, the local control plane determines whether a BFD controlpacket has been received on either an interface of the underlay controlchannel or an interface of the management control channel. An example ofthe BFD control packet is described in FIG. 5.

In step 406, the local control plane tests if such a BFD control packethas been received. If it has been, then the local control plane proceedsto performing step 408; otherwise, the local control plane proceeds toperforming step 430.

In step 408, the local control plane determines whether the received BFDcontrol packet was received via the underlay control channel. If it was,then the local control plane proceeds to performing step 410; otherwise,the local control plane proceeds to performing step 412.

In step 410, the local control plane parses a mandatory section of thedetected BFD control packet to determine whether any of certaindiagnostic codes are set in the mandatory section. The certaindiagnostic codes may include selected subset of diagnostic codes 0-31described in FIG. 5. The subset may include for example, a diagnosticcode “6” that indicates that a concatenated path to the second node isdown, and a diagnostic code “7” that indicates that the second nodeentered a maintenance mode, and thus became unavailable. The subset mayalso include other diagnostic codes described in FIG. 5.

Also, in this step, the local control plane updates the first stateusing the diagnostic code.

In step 414, the local control plane determines whether both the firststate and the second state indicate that the neighbor node isunreachable.

If the local control plane determines, in step 416, that the neighboringnode is unreachable, then the local control place proceeds to performingstep 418; otherwise, the local control plane proceeds to performing step420.

In step 418, the local control plane initiates a switchover of servicesfrom the neighbor node onto the node on which the local control plane isexecuted. Then, the local control plane proceeds to step 404.

In step 420, the local control plane determines whether any of the firststate and the second state indicates that the neighboring node is down.

If the local control plane determined, in step 422, that the neighboringnode is down, then the local control plane proceeds to step 418;otherwise, the local control plane proceeds to performing step 404.

In step 412, the local control plane parses the detected BFD controlpacket, extracts a diagnostic code from the packet, and uses thediagnostic code to update the second state. Then the local control planeproceeds to performing step 414, described above.

In step 430, the local control plane proceeds to performing step 450,described in FIG. 4B.

FIG. 4B is an example flow chart for implementing a two-channelhigh-availability approach.

In step 450, the local control plane determines if a timeout for waitingfor a BFD control message from the underlay control channel has expired.

If the local control plane determined, in step 452, that the timeout hasexpired, then the local control plane proceeds to performing step 454;otherwise, the local control plane proceeds to performing step 456.

In step 454, the local control plane sets the first state to indicatethat the neighboring node is unreachable.

In step 462, the local control plane proceeds to performing step 414.

In step 456, the local control determines if a timeout for waiting for aBFD control message from the management control channels has expired.

If the local control plane determined, in step 458, that the timeout hasexpired, then the local control plane proceeds to performing step 460,described above; otherwise, the local control channel proceeds toperforming step 464.

In step 464, the local control plane proceeds to performing step 404,described in FIG. 4A.

The process described in FIG. 4A-4B may be repeated for each type ofdiagnostic codes that the local control plane is implemented toconsider. The process may also be repeated for each node in a clusterwith which the first node is able to establish both an underlay controlchannel and a management control channel.

5. Example Diagnostic Codes Used in Two-Channel-Based HA Approach

FIG. 5 is a block diagram depicting an example mandatory section 520 ofan example of a generic BFD control packet 500. Generic BFD controlpacket 500 has a mandatory section 520, and an optional authenticationsection 530. If authentication section 530 is present, then the formatof authentication section 530 depends on the type of authentication inuse. Authentication section 530 is outside of the scope of thisdisclosure.

Mandatory section 520 of BFD control packet 500 includes a version field502, a diagnostic field 504, a state field 506, a P-F-C-A-D-M flag field508, a detection time multiplier field 510, a BFD control packet lengthfield 512, and other fields. Diagnostic field 504 is relevant for thisdisclosure, and therefore it is described in detail below.

Diagnostic field 504 includes five bits, and the bits are used to encodediagnostic codes. In an embodiment, the diagnostic codes include: 0—nodiagnostic, 1—control detection time expired, 2—echo function failed,3—neighbor signaled session down, 4—forwarding plane reset, 5—path down,6—concatenated path down, 7—administratively down, 8—reverseconcatenated path down, 9-31—reserved for future use.

In an embodiment, a diagnostic code “6” and a diagnostic code “7” areused in a two-channel-based HA approach. A local control plane, or anentity detecting a problem with a node, sends a BFD control packet witha diagnostic code “6” set if a northbound routing goes down, and thus aconcatenated path to, or via, the node is down. A local control plane,or an entity detecting a problem with the node, sends a BFD controlpacket with a diagnostic node “7” set if the node enters for example amaintenance mode, and the node is down by an administrator.

In an embodiment, other diagnostic codes, such as some codes of thereserved 9-31 codes, may be used in implementing a two-channel-based HAapproach.

Diagnostic codes included in BFD control packets may be used todetermine state of a node. In a mapping 550, a diagnostic code “0”indicates that a node is operational, while diagnostic codes “6”-“7”indicate that a node is down. If no BFD control message is received onboth channels before a timeout, then a node is considered to beunreachable.

6. Improvements Provided by Certain Embodiments

In an embodiment, an approach provides mechanisms for atwo-channel-based HA in a cluster of nodes for detecting failures ofnodes efficiently and reliably. The approach allows reducing, if noteliminating, false detections of node failures, and unnecessaryfailovers in the clusters.

In an embodiment, two-channel-based HA relies on communicationsexchanged via two channels established between hosts hosting nodes of acluster. The two channels provide support for the BFD-basedcommunications. Local control planes implemented in the hosts hostingthe nodes monitor BFD control packets exchanged via both channels. TheBFD control packets may include diagnostic codes that indicate status orproblems with the nodes. Based on the diagnostic codes, the localcontrol planes may determine whether failover is necessary.

7. Implementation Mechanisms

The present approach may be implemented using a computing systemcomprising one or more processors and memory. The one or more processorsand memory may be provided by one or more hardware machines. A hardwaremachine includes a communications bus or other communication mechanismsfor addressing main memory and for transferring data between and amongthe various components of hardware machine. The hardware machine alsoincludes one or more processors coupled with the bus for processinginformation. The processor may be a microprocessor, a system on a chip(SoC), or other type of hardware processor.

Main memory may be a random-access memory (RAM) or other dynamic storagedevice. It may be coupled to a communications bus and used for storinginformation and software instructions to be executed by a processor.Main memory may also be used for storing temporary variables or otherintermediate information during execution of software instructions to beexecuted by one or more processors.

8. General Considerations

Although some of various drawings may illustrate a number of logicalstages in a particular order, stages that are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings may be specifically mentioned, others willbe obvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed regarding specific embodiments. However, the illustrativeembodiments above are not intended to be exhaustive or to limit thescope of the claims to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen to best explain the principles underlying theclaims and their practical applications, to thereby enable othersskilled in the art to best use the embodiments with variousmodifications as are suited to the uses contemplated.

Any definitions set forth herein for terms contained in the claims maygovern the meaning of such terms as used in the claims. No limitation,element, property, feature, advantage or attribute that is not expresslyrecited in a claim should limit the scope of the claim in any way. Thespecification and drawings are to be regarded in an illustrative ratherthan a restrictive sense.

As used herein the terms “include” and “comprise” (and variations ofthose terms, such as “including,” “includes,” “comprising,” “comprises,”“comprised” and the like) are intended to be inclusive and are notintended to exclude further features, components, integers or steps.

References in this document to “an embodiment,” indicate that theembodiment described or illustrated may include a particular feature,structure, or characteristic, but every embodiment may not necessarilyinclude the particular feature, structure, or characteristic. Suchphrases are not necessarily referring to the same embodiment. Further,when a particular feature, structure, or characteristic is described orillustrated in connection with an embodiment, it is believed to bewithin the knowledge of one skilled in the art to affect such feature,structure, or characteristic in connection with other embodimentswhether or not explicitly indicated.

Various features of the disclosure have been described using processsteps. The functionality/processing of a given process step couldpotentially be performed in different ways and by different systems orsystem modules. Furthermore, a given process step could be divided intomultiple steps and/or multiple steps could be combined into a singlestep. Furthermore, the order of the steps can be changed withoutdeparting from the scope of the present disclosure.

It will be understood that the embodiments disclosed and defined in thisspecification extend to alternative combinations of the individualfeatures and components mentioned or evident from the text or drawings.These different combinations constitute various alternative aspects ofthe embodiments.

What is claimed is:
 1. A method for providing two-channel-basedhigh-availability in a node cluster, the method comprising: initiating,by a local control plane executing on a first node, a first state for anunderlay control channel and a second state for a management controlchannel; detecting, by the local control plane, a bidirectionalforwarding detection (“BFD”) control packet from a second node;determining, by the local control plane, whether the BFD control packethas been received from the underlay control channel; in response todetermining that the BFD control packet was received from the underlaycontrol channel: parsing, by the local control plane, the BFD controlpacket to extract a first diagnostic code; updating the first state withthe first diagnostic code; determining whether both the first state andthe second state indicate that the second node is unreachable; and inresponse to determining that both the first state and the second stateindicate that the second node is unreachable, initiating a switchover ofservices configured on the second node.
 2. The method of claim 1,further comprising: in response to determining that the BFD controlpacket was not received from the underlay control channel: determining,by the local control plane, whether the BFD control packet has beenreceived from the management control channel; in response to determiningthat the BFD control packet was received from the management controlchannel: parsing, by the local control plane, the BFD control packet toextract a second diagnostic code; updating the second state with thesecond diagnostic code; determining whether both the first state and thesecond state indicate that the second node is unreachable; and inresponse to determining that both the first state and the second stateindicate that the second node is unreachable, initiating a switchover ofservices configured on the second node.
 3. The method of claim 2,further comprising: in response to determining that both the first stateand the second state do not indicate that the second node isunreachable: determining whether any of the first state and the secondstate indicates that the second node is down: in response to determiningthat any of the first state and the second state indicates that thesecond node is down, initiating a switchover of services configured onthe second node.
 4. The method of claim 3, further comprising: inresponse to determining that no BFD control packet has been receivedfrom the underlay control channel before a timeout, update the firststate to indicate that the second node is unreachable.
 5. The method ofclaim 4, further comprising: in response to determining that no BFDcontrol packet has been received from the management control channelbefore a timeout, update the second state to indicate that the secondnode is unreachable.
 6. The method of claim 5, wherein the underlaycontrol channel is established between a first virtual network interfacecard (“VNIC”) configured on the first node and a first VNIC configuredon the second node; and wherein a management control channel establishedbetween a second VNIC configured on the first node and a second VNICconfigured on the second node.
 7. The method of claim 6, wherein theunderlay control channel is established between a first physical networkinterface card (“PNIC”) configured on the first node and a first PNICconfigured on the second node; and wherein the management controlchannel established between a second PNIC configured on the first nodeand a second PNIC configured on the second node.
 8. One or morenon-transitory computer-readable storage media storing one or morecomputer instructions which, when executed by one or more processors,cause the one or more processors to provide two-channel-basedhigh-availability in a node cluster, and to perform: initiating, by alocal control plane executing on a first node, a first state for anunderlay control channel and a second state for a management controlchannel; detecting, by the local control plane, a bidirectionalforwarding detection (“BFD”) control packet from a second node;determining, by the local control plane, whether the BFD control packethas been received from the underlay control channel; in response todetermining that the BFD control packet was received from the underlaycontrol channel: parsing, by the local control plane, the BFD controlpacket to extract a first diagnostic code; updating the first state withthe first diagnostic code; determining whether both the first state andthe second state indicate that the second node is unreachable; and inresponse to determining that both the first state and the second stateindicate that the second node is unreachable, initiating a switchover ofservices configured on the second node.
 9. The one or morenon-transitory computer-readable storage media of claim 8, comprisingadditional instructions which, when executed by the one or moreprocessors, cause the one or more processors to perform: in response todetermining that the BFD control packet was not received from theunderlay control channel: determining, by the local control plane,whether the BFD control packet has been received from the managementcontrol channel; in response to determining that the BFD control packetwas received from the management control channel: parsing, by the localcontrol plane, the BFD control packet to extract a second diagnosticcode; updating the second state with the second diagnostic code;determining whether both the first state and the second state indicatethat the second node is unreachable; and in response to determining thatboth the first state and the second state indicate that the second nodeis unreachable, initiating a switchover of services configured on thesecond node.
 10. The one or more non-transitory computer-readablestorage media of claim 9, comprising additional instructions which, whenexecuted by the one or more processors, cause the one or more processorsto perform: in response to determining that both the first state and thesecond state do not indicate that the second node is unreachable:determining whether any of the first state and the second stateindicates that the second node is down: in response to determining thatany of the first state and the second state indicates that the secondnode is down, initiating a switchover of services configured on thesecond node.
 11. The one or more non-transitory computer-readablestorage media of claim 10, comprising additional instructions which,when executed by the one or more processors, cause the one or moreprocessors to perform: in response to determining that no BFD controlpacket has been received from the underlay control channel before atimeout, update the first state to indicate that the second node isunreachable.
 12. The one or more non-transitory computer-readablestorage media of claim 11, comprising additional instructions which,when executed by the one or more processors, cause the one or moreprocessors to perform: in response to determining that no BFD controlpacket has been received from the management control channel before atimeout, update the second state to indicate that the second node isunreachable.
 13. The one or more non-transitory computer-readablestorage media of claim 12, wherein the underlay control channel isestablished between a first virtual network interface card (“VNIC”)configured on the first node and a first VNIC configured on the secondnode; and wherein a management control channel established between asecond VNIC configured on the first node and a second VNIC configured onthe second node.
 14. The one or more non-transitory computer-readablestorage media of claim 13, wherein the underlay control channel isestablished between a first physical network interface card (“PNIC”)configured on the first node and a first PNIC configured on the secondnode; and wherein the management control channel established between asecond PNIC configured on the first node and a second PNIC configured onthe second node.
 15. A local control plane implemented in a hostcomputer and configured to provide two-channel-based high availabilityin a node cluster, the local control plane comprising: one or moreprocessors; one or more memory units; and one or more non-transitorycomputer-readable storage media storing one or more computerinstructions which, when executed by the one or more processors, causethe one or more processors to perform: initiating, by a local controlplane executing on a first node, a first state for an underlay controlchannel and a second state for a management control channel; detecting,by the local control plane, a bidirectional forwarding detection (“BFD”)control packet from a second node; determining, by the local controlplane, whether the BFD control packet has been received from theunderlay control channel; in response to determining that the BFDcontrol packet was received from the underlay control channel: parsing,by the local control plane, the BFD control packet to extract a firstdiagnostic code; updating the first state with the first diagnosticcode; determining whether both the first state and the second stateindicate that the second node is unreachable; and in response todetermining that both the first state and the second state indicate thatthe second node is unreachable, initiating a switchover of servicesconfigured on the second node.
 16. The local control plane of claim 15,storing additional instructions which, when executed by the one or moreprocesses, cause the one or more processors to perform: in response todetermining that the BFD control packet was not received from theunderlay control channel: determining, by the local control plane,whether the BFD control packet has been received from the managementcontrol channel; in response to determining that the BFD control packetwas received from the management control channel: parsing, by the localcontrol plane, the BFD control packet to extract a second diagnosticcode; updating the second state with the second diagnostic code;determining whether both the first state and the second state indicatethat the second node is unreachable; and in response to determining thatboth the first state and the second state indicate that the second nodeis unreachable, initiating a switchover of services configured on thesecond node.
 17. The local control plane of claim 16, storing additionalinstructions which, when executed by the one or more processes, causethe one or more processors to perform: in response to determining thatboth the first state and the second state do not indicate that thesecond node is unreachable: determining whether any of the first stateand the second state indicates that the second node is down: in responseto determining that any of the first state and the second stateindicates that the second node is down, initiating a switchover ofservices configured on the second node.
 18. The local control plane ofclaim 17, storing additional instructions which, when executed by theone or more processes, cause the one or more processors to perform: inresponse to determining that no BFD control packet has been receivedfrom the underlay control channel before a timeout, update the firststate to indicate that the second node is unreachable.
 19. The localcontrol plane of claim 18, storing additional instructions which, whenexecuted by the one or more processes, cause the one or more processorsto perform: in response to determining that no BFD control packet hasbeen received from the management control channel before a timeout,update the second state to indicate that the second node is unreachable.20. The local control plane of claim 19, wherein the underlay controlchannel is established between a first virtual network interface card(“VNIC”) configured on the first node and a first VNIC configured on thesecond node; and wherein a management control channel establishedbetween a second VNIC configured on the first node and a second VNICconfigured on the second node.